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SIGNAL PROCESSING APPARATUS. METHOD OF THE SAME. 
AN IMAGE PROCESSING APPARATUS AND METHOD OF THE SAME 

5 BACKGROUND OF THE INVENTION 

1. Field of the Invention 

The present invention relates to a signal 
processing apparatus and method using a difference in 
sampling phase for removing aliasing components and to 
10 thereby broaden the frequency band of a signal and an 

image processing apparatus and method for processing an 
image signal obtained by capturing a plurality of images 
having a predetermined difference in sampling phase by a 
single imaging element. 
15 2. Description of the Related Art 

In the related art, there has been a signal 
processing method in a charged coupled device (CCD) or 
other image sensor for obtaining a broad band image 
doubled in the number of pixels by inputting two images 
20 shifted by a pitch of exactly half of the sampling 

interval and up- sampling and adding the two. The signal 
processing method is applicable to a black and white CCD 
image sensor, triple -element RGB CCD image sensor, etc. 

Also, there have been attempts to apply this to 
2 5 an image obtained by a single -element CCD image sensor 
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having a color filter, currently the mainstream in video 
cameras and still image pickup devices (hereinafter also 
referred to as electronic still cameras). Since a single- 
element CCD image sensor has a color filter cycle of, for 
5 example, every two pixels etc., it has been considered to 
shift images by half of the cycle, that is, one pixel. A 
signal of an image captured by a single- element CCD image 
sensor and converted from an analog to digital format is 
processed in a later camera signal processor to generate 

10 an RGB signal or a luminance signal and a color 

difference signal using interpolation for each color or 
an interrelationship among the colors. 

Summarizing the problem to be solved by the 
invention, in a single- element CCD image sensor, however, 

15 when shifting the pixels, the array of signals of the 

colors becomes different from the conventional array due 
to the increase in the number of pixels. The later camera 
signal processor therefore not only has to process an 
increased number of pixels, but also has to change the 

20 algorithm itself. 

Also, when shifting the pixels by less than the 
pixel pitch of the CCD image sensor, there is the 
disadvantage that the camera signal processor has to 
process a signal with uneven sampling, so the processing 

2 5 becomes complex. 
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For example, when using a Bayer array single- 
element CCD Image sensor obtaining a R (red), G (green), 
and B (blue) pixel array pattern shown In Fig. 12A to 
obtain a total of four Images shifted one pixel each In 
5 the vertical, horizontal, and diagonal directions, the 

obtained pixel arrays of the R, G, B data become as shown 
in Figs. 12E, 12F, and 12G, which are different from the 
normal pixel arrays shown in Figs. 12B, 12C, and 12D. 

Namely, as will be understood by comparing 
10 Figs. 12C and 12D with Figs. 12F and 12G, the numbers of 
samples of R data and B data are increased four times, 
while as is understood by comparing Fig. 12B and Fig. 
12E, the number of samples of G data is only doubled. 

Therefore, the camera signal processor has to 
15 process R, G, B data of the pixel arrays shown in Figs. 

12E, 12F, and 12G, so there is the disadvantage that the 
processing becomes complex. 

Also, when shifting an image by less than the 
pixel pitch, the seunpling intervals become uneven, so 
20 there is the disadvantage that the processing becomes 
difficult . 



25 



SUMMARY OF THE INVENTION 

An object of the present invention is to provide an 
image processing apparatus and method capable of 



obtaining a high resolution Image from a low resolution 
Image without making the processing In the camera signal 
processor complex and a signal processing apparatus and 
method using the band broadening method used In the image 
5 processing apparatus etc. 

To attain the above object, according to a first 
aspect of the present invention, there is provided a 
signal processing apparatus comprising a transforming 
means for transforming a plurality of first digital 

10 signals having mutually different sampling phases to 
generate a plurality of second digital signals In 
frequency domain; a memory means for storing a plurality 
of complex numbers corresponding to the sampling phases; 
and a processing means for multiplying the plurality of 

15 second digital signals and the plurality of complex 

numbers corresponding to the plurality of second digital 
signals and adding the multiplication results to generate 
a third digital signal free from aliasing components. 

In the signal processing apparatus of a first aspect 

20 of the present invention, the transforming means 

transforms the plurality of first digital signals having 
mutually different sampling phases a plurality of second 
digital signals in the frequency domain. 

Then, the processing means multiplies the plurality 

2 5 of second digital signals with a plurality of complex 



numbers stored in the memory means corresponding to the 
plurality of second digital signals and adds the 
multiplied results to generate third digital signals 
freed from aliasing components. 

Preferably, the apparatus further comprises a phase 
shift means for shifting the phase of the first digital 
signals or the second digital signals by exactly a 
predetermined phase. 

According to a second aspect of the present 
invention, there is provided a signal processing 
apparatus, comprising a phase shift means for shifting 
the phase of a plurality of first digital signals having 
mutually different sampling phases to generate second 
digital signals; a memory means for storing a plurality 
of real numbers indicating real parts and imaginary parts 
of a plurality of predetermined complex numbers 
corresponding to the sampling phases; a processing means 
for multiplying the first digital signals with real 
numbers indicating the real parts corresponding to the 
first digital signals to obtain first multiplied results, 
multiplying the second digital signals corresponding to 
the first digital signals with real numbers indicating 
the imaginary parts corresponding to the second digital 
signals to obtain second multiplied results, and adding 
the first multiplied results and the second multiplied 



results to generate third digital signals free of 
aliasing components - 

According to a third aspect of the present 
invention, there is provided an image processing 
apparatus, comprising an image input means for generating 
a plurality of first image signals having mutually 
different sampling phases in accordance with imaging 
results; a transforming means for transforming the first 
image signals to a frequency domain to generate a 
plurality of second image signals; a memory means for 
storing a plurality of complex numbers corresponding to 
the sampling phases; a processing means for multiplying 
the plurality of second image signals with the plurality 
of complex numbers corresponding to the plurality of 
second image signals and adding the multiplied results to 
generate third image signals free from aliasing 
components . 

Preferably, the image input means forms an image of 
a plurality of color lights passed through a single-plate 
type color filter on corresponding pixels among a 
plurality of pixels arranged in a matrix two- 
dimensionally to generate the first image signals 
comprised by color data of the plurality of colors; and 
the processing means performs the multiplioation and the 
addition for every color data of the plurality of color 
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data to generate a plurality of fourth image signals 
corresponding to the plurality of colors and generates 
the third image signals by using the plurality of fourth 
image signals. 

5 More preferably, the sampling phase is determined 

for a predetermined one color among the plurality of 
colors in order that a sampling pattern of color data of 
the color included in the plurality of first image 
signals and a sampling pattern of color data included in 

10 the fourth image signals of the color become similar. 

Preferably, the processing means comprises a spatial 
shift means for spatially shifting the second image 
signals in accordance with the sampling phases; a basic 
spectrum calculation means for multiplying the spatially 

15 shifted plurality of second image signals with the 
plurality of complex numbers corresponding to the 
plurality of second image signals and adding the 
multiplied results to calculate a basic spectrum free 
from aliasing components; and an inverse transforming 

20 means for transforming the basic spectrum from a 

frequency domain to a time domain to generate the third 
image signals. 

Alternatively, preferably the apparatus further 
comprises a drive means for moving the image input means 

25 physically, optically, or electrically so that the image 



input means can generate a plurality of image signals 
having mutually different sampling phases in accordance 
with the imaging results. 

Alternatively, more preferably the image input means 
5 is a single -element CCD image sensor and the color filter 
is a primary color filter or a color compensation filter. 

According to a fourth aspect of the present 
invention, there is provided an image processing 
apparatus, comprising an image input means for receiving 

10 as input a plurality of first image signals having 

mutually different sampling phases in accordance with 
imaging results; a phase shift means for shifting the 
phase of the plurality of first digital signals to 
generate second digital signals; a memory means for 

15 storing a plurality of real numbers respectively 

indicating real parts and imaginary parts of a plurality 
of predetermined complex numbers corresponding to the 
sampling phases; a processing means for multiplying the 
first digital signals with real numbers indicating the 

20 real parts corresponding to the first digital signals to 
obtain first multiplication results, multiplying the 
second digital signals corresponding to the first digital 
signals with real numbers indicating the imaginary parts 
corresponding to the second digital signals to obtain 

2 5 second multiplied results, and adding the first 



multiplied results and the second multiplied results to 
generate third digital signals free from aliasing 
components . 

According to a fifth aspect of the present 
5 invention, there is provided a signal processing method 
comprising transforming a plurality of first digital 
signals having mutually different sampling phases to a 
frequency domain to generate a plurality of second 
digital signals; multiplying the plurality of second 

10 digital signals with a plurality of complex numbers 

corresponding to the plurality of second digital signals 
and adding the multiplied results to generate third 
digital signals free from aliasing components. 

According to a sixth aspect of the present 

15 invention, there is provided a signal processing method 
using a plurality of real numbers indicating real parts 
and imaginary parts of a predetermined plurality of 
complex numbers corresponding to sampling phases, 
comprising shifting a plurality of first digital signals 

20 having mutually different sampling phases by 

predetermined phases to generate second digital signals; 
multiplying the first digital signals with real numbers 
indicating the real parts corresponding to the first 
digital signals to generate first multiplied results; 

25 multiplying the second digital signals corresponding to 
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the first digital signals with real numbers indicating 
the imaginary parts corresponding to the second digital 
signal to generate second multiplied results; and adding 
the first multiplied results and the second multiplied 
5 results to generate third digital signals free of 
aliasing components. 

According to a seventh aspect of the present 
invention, there is provided an image processing method 
comprising generating a plurality of first image signals 

10 having mutually different sampling phases in accordance 
with imaging results; converting the first image signals 
to a frequency domain to generate a plurality of second 
image signals; and multiplying the plurality of second 
image signals with a plurality of complex numbers 

15 corresponding to the plurality of second image signals 
and adding the multiplication results to generate third 
image signals free from aliasing components. 

Preferably, the method further comprises forming an 
image of a plurality of color lights passing through a 

20 single -plate type color filter on corresponding pixels 
among a plurality of pixels arranged in a matrix two- 
dimensionally to generate the first image signals 
comprised by color data of the plurality of colors and 
performing the multiplication and the addition for every 

25 color data of the plurality of colors to generate a 



plurality of fourth image signals corresponding to the 
plurality of colors and generate the third image signals 
by using the plurality of fourth image signals. 

More preferably, the method further comprises 
5 determining the sampling phase for a predetermined one 
color among the plurality of colors so that a sampling 
pattern of color data of the color included in the 
plurality of first image signals and a sampling pattern 
of color data included in the fourth image signals of the 

10 color become similar. 

According to an eighth aspect of the present 
invention, there is provided an image processing method 
using a plurality of real numbers prepared in advance 
indicating real parts and imaginary parts of a 

15 predetermined plurality of complex numbers corresponding 
to sampling phases, comprising generating a plurality of 
first image signals having mutually different sampling 
phases in accordance with imaging results; shifting by a 
predetermined phase the plurality of first digital 

20 signals to generate second digital signals; multiplying 
the first digital signals with real numbers indicating 
the real parts corresponding to the first digital signals 
to generate first multiplied results; multiplying the 
second digital signals corresponding to the first digital 

2 5 signals with real numbers indicating the imaginary parts 
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corresponding to the second digital signals to generate 
second multiplied results; and adding the first 
multiplied results and the second multiplied results to 
generate third digital signals free of aliasing 
5 components . 

BRIEF DESCRIPTION OF THE DRAWINGS 

These and other objects and features of the present 
invention will become clearer from the following 
description of the preferred embodiments given with 
10 reference to the accompanying drawings, in which: 

Fig. 1 is a view for explaining an RGB Bayer array 
single-element CCD image sensor used in an electronic 
still camera of a first embodiment of the present 
invention; 

15 Fig. 2 is a view of a part of the configuration of 

the electronic still camera of the first embodiment of 
the present invention; 

Fig. 3 is a view for explaining a sampling phase of 
four images captured by a CCD image sensor shown in Fig. 
20 2 for obtaining one high resolution image; 

Fig. 4 is a view for explaining processing in a 
signal processor shown in Fig. 2; 

Fig. 5A is a view for explaining a sampling phase of 
R data to be processed in a signal processor shown in 
2 5 Fig. 2 and Fig. 5B is a view for explaining a sampling 
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phase of B data to be processed ±n tlie signal processor 
shown in Fig. 2; 

Fig. 6 is a view of the configuration of the signal 
processor shown in Fig. 2; 
5 Fig. 7 is a view of a part of the configuration of 

an electronic still camera of a second embodiment of the 
present invention; 

Fig. 8 is a view of the configuration of a signal 
processor shown in Fig. 7; 
10 Fig. 9 is a view of a part of the configuration of 

an electronic still camera of a third embodiment of the 
present invention; 

Fig. 10 is a view for explaining a color 
compensating filter used in the CCD image sensor shown in 
15 Fig. 9; 

Fig. 11 is a view of the configuration of a signal 
processor shown in Fig. 9; and 

Fig. 12 is a view for explaining disadvantages of 
the related art. 

20 

DESCRIPTION OF THE PREFERRED EMBODIMENTS 
Below, an electronic still camera according to the 
present embodiments will be explained. 
First Embodiment 
2 5 An electronic still camera of the present embodiment 
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is provided with an RGB Bayer array single-mode CCD image 
sensor and processes four sets of RGB data different in 
sampling phases obtained by the CCD image sensor using a 
predetermined band broadening method to increase the 
5 resolution and thereby obtain RGB data having a double 
bandwidth corresponding to double the number of pixels 
vertically and horizontally. 

Specifically, the electronic still camera obtains G 
data, R data, and B data of the pixel arrays shown in 
10 Figs. IB, IC, and ID from the four sets of RGB data 

different in sampling phase, including the RGB data of 
the pixel array shown in Fig. lA, and add these to obtain 
the RGB data of the pixel array shown in Fig. IE. 

Figure 2 is a view of a part of the configuration of 
15 an electronic still camera of the present embodiment. 

As shown in Fig. 2, the electronic still camera 1 
comprises a lens 2, an optical LPF 3, a CCD image sensor 
4, an AD converter 5, a CCD drive 6, a memory 7, a signal 
processor 8, and a camera signal processor 9. 
20 [Optical LPF 3] 

The optical LPF 3 is an optical low pass filter 
(LPF) for removing unnecessary high band components from 
light from the object being photographed passing through 
the lens 2. 
25 [CCD 4] 



- 15 - 



The CCD image sensor 4 comprises a plurality of 
photodiodes arranged in a matrix and a Bayer color filter 
array. The Bayer color filter array is comprised of G 
(green) color filters arranged in a checkerboard pattern 
5 and R (red) filters and B (blue) filters arranged in a 
checkerboard pattern in the remaining parts. The 
photodiodes receive light passing through the Bayer color 
filter array and convert the light to electrical signals 
to generate received light signals S4o to 4 in 

10 accordance with the amount of received light and outputs 
the received light signals S4o to S43 to the AD 
converter. 

The CCD image sensor 4 is, as will be explained 
later on, driven to move by the CCD drive 6 and captures 
15 four images different in sampling phases for obtaining a 
single high resolution image. 
[AD Converter 5] 

The AD converter 5 converts the received light 
signals S4(, to S43 to image signals S5o to 863 and outputs 
20 an image signal S5 to the signal processor 8. 
[CCD Drive 6] 

The CCD drive 6 physically moves the CCD image 
sensor 4 by exactly a predetermined distance in a 
predetermined direction based on a control signal S8a 
2 5 from the signal processor 8 so that the CCD image sensor 
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4 captures a total of four Images respectively having 
phases of (0, 0), (n/2, -n/2), (0, n), (n/2, n/2) when 
assuming sampling intervals of the CCD image sensor 4 to 
be 2n and the phase of one image to be the origin (0, 0) 
5 as shown in Fig. 3. 

The signal processor 8 uses the image signal S5 of 
the four captured images having mutually different 
sampling phases as shown in Fig. 3 input from the AD 
converter 5 to generate an image signal S8b of an image 

10 having double the number of pixels in the vertical and 
horizontal directions. 

Here, the image signal S5 is composed of the pixel 
data of pixels arranged in the matrix constituting the 
CCD image sensor 4 . Pixel data of pixels at positions 

15 indicated by "G" in Fig. 4A indicate G (green) data, 

pixel data of pixels at the positions indicated by "R" in 
Fig. 4B indicate R (red) data, and pixel data of pixels 
at the positions indicated by "B" in Fig. 4C indicate B 
(blue) data. 

20 The signal processor 8 writes the image signal S5 of 

the four captured images input from the AD converter 5 in 
the memory 7 , then reads it out and processes the image 
signal S5 by a later explained band broadening method to 
generate an image signal S8b so as to generate R data and 

25 B data of the pixel array patterns shown in Figs. IC and 
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ID from R data and B data of the pixel array patterns 
shown in Figs. 4B and 4C. 

Note that the pixel array pattern of G data shown in 
Fig. 4A and the pixel array pattern of G data shown in 
5 Fig. IB are the same, so no processing is necessary for 
the G data. Namely, the signal processor 8 uses the G 
data included in the image signal S5 as it is as the G 
data of the image signal S8b. 

Also, the signal processor 8 generates a control 

10 signal SBa based on information on the direction of 

movement and distance of the CCD image sensor 4 stored in 
the memory 7 in advance so as to obtain four captured 
images having mutually different sampling phase shown in 
Fig. 3 and outputs the control signal S8a to the CCD 

15 drive 6. 

Below, the processings for generating R data and B 
data of the image signal S8b in the signal processor 8 
will be explained. 

Note that the processings for generating R data and 
20 B data are the same except for the value of the sampling 
phase. 

Here, the "sampling phase" indicates the coordinates 
of a pixel corresponding to pixel data of a certain color 
included in the image signal S5 of the four captured 
2 5 images when assuming the position of a certain pixel 
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corresponding to p±xel data of that color included in the 
image signal S8b to be the origin. 

Specifically, for the R data, the sampling phases of 
the four captured images, as shown in Fig. 5A, 
5 respectively become (a^o' ayo) = (n/2, 0), {cc^i' ayi) = {n, - 
n/2), (a^2' ay2) = {n/2, n), and (a^g- ay3) = (n, n/2). 

Also, for the B data, the SEimpling phases of the 
four captured images, as shown in Fig. 5B, respectively 
become (a^o' ayo) = {0, n/2), (a^i' ayi) = (n/2, 0), (a^2 ay2) = 
10 (0, 3n/2), and (cx^g- (x^j) = {n/2, n) . 

The values of these sampling phases are used as 
(a^o' ocyo) to (a^3' ttyg) at the time of the later explained 
spatial shifting and are stored in the memory 7 shown in 
Fig. 2 in advance. 
15 Figure 6 is a view of the configuration of the 

signal processor 8 . 

As shown in Fig. 6, the signal processor 8 comprises 
an interpolation circuit 50, a Fourier transform circuit 
51, a spatial shift circuit 52, a basic spectrum 
20 calculation circuit 53, and an inverse Fourier transform 
circuit 54. 

The interpolation circuit 50 receives as inputs 
image signals S5o to SSj of the four captured images to 
obtain one high resolution image, interpolates these and 
2 5 up- samples them two-fold to generate image signals S50o 
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to S5O3, and outputs the image signals SSOq to S5O3 to the 
Fouriei: ti-ansform circuit 51- 

Speoif ioally, the interpolation circuit 50 inserts 
(M^-1) number of "O's" between the sampling positions in 
the X direction and (My-l) number of "O's" between the 
sampling positions in the y direction for the image 
signals S5o to S53 to generate the image signals S50o to 
S5O3. 

Note that and My are multiples indicating how 
many multiples of frequency components for restoration to 
in the x direction and y direction with respect to the 
Nyquist frequency. In the present embodiment, both are 
"2" . 

The Fourier transform circuit 51 converts the image 
signals S50o to S5O3 input from the interpolation circuit 
50 to generate image signals S51o to S5I3 of a frequency 
domain expression and outputs the image signals SSIq to 
S5I3 to the spatial shift circuit 52. 

The spatial shift circuit 52 performs two- 
dimensional spatial shifting on the R data and B data 
included in the image signals S51o to S5I3 obtained by 
imaging to generate image signals to Y3 to express 
that sampling was performed in the positions of Figs. 4B 
and 4C. 

Namely, the spatial shift circuit 52 multiplies R 
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data and B data of the Image signals S51o to SSlg with 
exp(-3(oo^ of^o+«y ayo)/(2n)) to exp(-j{oL)^ «^3+"y ay3)/(2n)), 
respectively, to generate image signals Yo to Y3. 

Note that, at this time, the portion above the 
5 Nyquist frequency indicates negative frequencies both in 
the X direction and y direction. The cox or coy in this 
portion has to be made (Wx-cOg^) or (cOy-cOgy) . Here, and 
cOgy respectively are sampling frequencies in the x 
direction and y direction before up- sampling. 

10 The basic spectrum calculation circuit 53 uses the 

image signals ¥□ to Y3 input from the spatial shift 
circuit 52 and complex numbers to W3, w,,' to W3 ' stored 
in the memory 7 shown in Fig. 2 to separately generate 
the two-dimensional image signals Xg^ of the R data and B 

15 data and outputs the image signal S53 comprised of the 

generated image signals Xoo of the R data and B data and 
the G data of a pixel in the position shown in Fig. 4A to 
the inverse Fourier transform circuit 54. 

Note that the memory 7 stores, for each of the R 

20 data and B data generated as explained later, the complex 
numbers Wq to W3 to be multiplied with the image signals 
Yo to Y3 for finding the basic spectrum components X13o,o 
in the first and third quadrants and the complex numbers 
Wo' to W3' to be multiplied with the image signals Yq to 

25 Y3 for finding the basic spectrum components X24o,o of the 
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second and fourth quadrants. 

The basic spectrum calculation circuit 53 obtains 
basic spectrum components X13o „ not containing aliasing 
components based on the formula (1) below and obtains 
basic spectrum components X24o,o not containing aliasing 
components based on the formula (2) below i 

X13o,0 = ^{(1- 3)Yo +(1+ 3)Yl +(1- j)Y2 +{1+ j)Y3} 

. . .(1) 

Here, ( 1- j ) , { 1+ j ) , (1- j ) and (1+j) in the formula (1) 
are defined as Wo, w^, Wj and W3 respectively. 

^24o,o = 7{^l + J)YQ+(l-j)Y3_+(l + J)Y2 + (l- j)Y3} 

... (2) 

Here, ( 1+ j ) , ( 1- j ) , { 1+J ) and (1-j) in the formula (2) 
are defined as Wo', w^- , Wa' and W3' respectively. 

Then, as shown in formula (3) below, the basic 
spectrum components X13o,o SLnCL X24o 0 a^re added to generate 
the basic spectrum component Xq o- 

0,0 0,0 0,0 •■■K^) 

Below, the grounds of the above formulas (1) and (2) 
will be explained. 

First, the theory behind the band broadening method 
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using a plurality of discrete signals having differences 
in sampling phase in a two-dimensional direction 
constituting the grounds of the above formulas (1) and 
(2) will be explained. 

The variables are defined below: 

S total number of input digital signals (discrete 
signals) 

L indexes 0, 1, , S-1 of digital signals 

Y_orgi. L-th organization digital signal 

(frequency domain expression) 

Y—Orgj, L-th organization digital signal (spatial 
domain expression) 

Yl L-th organization digital signal spatially 
shifted (frequency domain expression) 

Yi. L-th organization digital signal spatially 
shifted (spatial domain expression) 
sampling phase difference (rad) in the x-axis 
direction with respect to 0th organization 
signal of L-th organization digital signal 
sampling phase difference (rad) in the y-axis 
direction with respect to 0th organization 
signal of L-th organization digital signal 
^. imaging components X(ciL)3j-i-cOgx» c«5y-i'«sy) of ^~ 

order in the x-direction and k-order in the y- 
direotion, basic spectrum when i=k=0 
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number of Imaging components having negative 
order in the x- direction 

number of imaging components having positive 
order in the x- direction 

Py number of imaging components having negative 
order in the y- direction 

Qy number of imaging components having positive 
order in the y- direction 
multiple indicating how many times of 
components with respect to Nyquist frequency 
were included in continuous signal input before 
being sampled in x- direction 

multiple indicating how many times of frequency 
components for restoration in the x- direction 
with respect to Nyquist frequency normally, 

Ny multiple indicating how many times of 

components with respect to Nyquist frequency 
were included in continuous signal input before 
being sampled in y-direotion 

multiple indicating how many times of frequency 
components for restoration in the y-direotion 
with respect to Nyquist frequency, normally, 

Ny=My 

Wi. complex number for multiplying with L-th 
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organization digital signal in frequency domain 



A signal with sampling phase shifted by exactly 
in the x- direction and ay in the y- direction with respect 
to a reference discrete signal is, when indicating the 
reference discrete signal by the formula (4) below, 
indicated by the formula (5) below. 

00 00 

Y(ft) .0) )= y. y. X(ta - ±oj o) -ko) ) ...(4) 
X ±jLoo k=-oo X sx y sy' '"^ ^ 



Y{coy^,Wy) = exp(j • (ft^x^x + - «y) / 2;r) • ( 

10 00 00 

1 1 exp(-j-(iax +kay))X(G;x -i^^sx-^^v -^^^sy) 
i=-ook = -oo 

...(5) 

Here, two-dimensional spatial shift can be attained 
by shifting to a correct position after interpolation 
when in the spatial domain - 
15 Also, when in the frequency domain, it is sufficient 

to multiply with "exp( - j (cox-ax+coyay ) / ( 2n ) ) " . 

The spatial shift circuit can be realized for both 
the spatial domain and frequency domain. 

For example, when performing two-dimensional spatial 
20 shift in the spatial domain, in Fig. 6, the Fourier 

transform circuit 51 and the spatial shift circuit 52 are 
switched. Namely, the Fourier transform is performed 
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after spatial shift. 

Next, consider a complex number simultaneous 
equation for obtaining a complex number to be 
multiplied with S number of discrete signals to reproduce 
5 basic spectrum components . 

When in one dimension, the fact that the spectrum 
becomes symmetric between a positive frequency and a 
negative frequency based on the characteristics of a 
Fourier transform is used. It is sufficient to consider 

10 only a positive frequency and the number of imaging 
components in the positive range. 

On the other hand, as in the present embodiment, 
when in two dimensions, due to the characteristics of a 
Fourier transform, conjugation occurs about the origin 

15 and not about the cox-axis or coy-axis. Namely, on a cox-coy 
plane, the first and third quadrants are conjugated and 
the second and the fourth quadrants are conjugated. 
Namely, the two quadrants next to each other, for 
example, the first quadrant and the second quadrant, are 

2 0 independent from each other. Accordingly, it is necessary 
to consider the imaging components included in two 
quadrants, for example, the first and second quadrants. 
Note that the remaining two quadrants can be obtained by 
using the conjugated relationship. 

25 Thus, let us consider the first and the second 
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quadrants. First, consider the first quadrant. Th.e number 
of discrete signals S necessary for obtaining solutions 
of a complex number simultaneous equation of the first 
quadrant becomes the product of the number of discrete 
5 signals necessary in the x- direction and the number of 
discrete signals necessary in the y-direction, that is, 

S=(P. + Qx+l) CPy + Qy+l) - 

Here, 

P^=FLoor( (N^-l)/2) 
10 Q^=FLoor( (N^+M^-l)/2) 

Py=FLoor( (Ny-l)/2) 
Qy=FLoor ( { Ny+My- 1 ) / 2 ) 
Note that in the present embodiment, when N^, Ny, M^, 
and My are all "2", S becomes "4". If obtaining four 
15 organizations of continuous signals, the imaging 

components can be removed and a signal up double the 
Nyquist frequency can be obtained. 

Accordingly, the complex number simultaneous 
equation can be indicated as the formula (6) below when 
20 the number of formulas and the unknown number of the 
basic spectrum components X^ ^ and imaging components 
are S and the discrete signal of the first and the third 
quadrants after spatial shifting is assumed to be ¥13^^^. 
By solving the formula (6) below to obtain the complex 
2 5 number Wj,, the basic spectrum components of the first and 



the third quadrants can be obtained as shown In the 
formula (7) below. Note that W^. in the formula (7) below 

indicates a complex number. 

Qx Qy 

YJ3,= V yesp(-j-(ia,,+ka,,))X,_, 

i^-Fx,k k Py 

where, L=0, 1, S-1 

... (6) 



Next , if considering the second quadrant , cOj^ is 
10 negative in the second quadrant and the imaging 

components in the x-direotion also becomes a negative 
order. Therefore, the complex number simultaneous 
equation can be indicated as in the formula (9) below 
when assuming the number of formulas and the unknown 
15 number of basic spectrum components Xg g and imaging 

components to be S and assuming the discrete signals 

in the second and fourth quadrants after spatial shifting 
to be Y24i,. By solving the formula (8) below to obtain 
the complex number Wj.', the basic spectrum components 
20 X24o,o can be obtained as shown in the formula (9) below. 

Note that Wj,' of the formula (9) below indicates a 
complex number different from Wj.. 
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= y y.exp(-j-(ia^j^+kayj,))X,j^ 

i Qxk Py 

where L=0, 1, .... S-1 

. . .(8) 



S-1 

X24,, = yw,'Y24, ---(9) 



The complex number w^, obtained from the above Is 
used for removing the Imaging components In the first and 
third quadrants, while the complex number Wl' Is used for 
removing the Imaging components In the second and the 
fourth quadrants . 

Here, since the processing for removing the imaging 
components in the first and the third quadrants and the 
processing for removing the imaging components of the 
second and the fourth quadrants are Independent, the 
signals are divided into two systems. 

The processing for dividing the components into two 
systems generates a signal where the values of the second 
and fourth quadrants are replaced by "0" in the frequency 
domain and a signal where the values of the first and 
third quadrants are replaced by "0" in a frequency 
domain. On the other hand, when in a spatial domain, it 
generates a signal filtered for extracting the frequency 



components of the second and the fourth quadrants and a 
signal filtered for extracting frequency components of 
the first and third quadrants. 

Multiplying the complex numbers and Wj,' 
5 respectively with the two organizations of signals gives 
the basic spectrum X13i. of the first and third quadrants 
wherein imaging components are removed and the basic 
spectrum S24l of the second and fourth quadrants wherein 
imaging components are removed. 
10 Finally, the basic spectrum X13i, and the basic 

spectrum X24i, from which the imaging components have been 
removed may be added to obtain the desired band broadened 
signal . 

This completes the explanation of the theory behind 
15 the band broadening method. 

Below, the grounds of the above formulas (1) and (2) 
will be explained based on the above theory of the band 
broadening method. 

Namely, a case where imaging components are removed 
20 from R data of pixels at positions shown in Fig. 4B, 

obtained from the image signal S5 of the four captured 
images , to broaden the band and R data of pixels at 
positions shown in Fig. IC are generated will be 
explained . 

2 5 Note that the case where imaging components are 
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removed from B data of pixels at positions shown in Fig- 
4C, obtained from the image signal S5 of the four 
captured images, to broaden the band and B data of pixels 
at positions shown in Fig. ID are generated is the same 
5 as the case of R data after removal of values of the 
sampling phase differences a^t- ^yL' 

In the first and third quadrants, the formulas (10a) 
to (lOd) are obtained from the above formula (6). 



yo=Xo,o-3Xi,o+Xo,i-jXi., . . . (10a) 

Y,=Xo.o-Xi,o+ jXo,i- jXi., . . . ( 10b) 

Y2=Xo,o- jXi.o-Xo,i+jXi,3. . . . (lOo) 

y3=Xo.o-Xi.o- jXo,i+jXi.i . . . (lOd) 



15 When solving the above formulas (10a) to (lOd), the 

above formula ( 1 ) corresponding to the above formula ( 7 ) 
is introduced and the basic spectrum X13o,o of the first 
and third quadrants is obtained. 

Regarding the second and fourth quadrants, the 

20 formulas (11a) to (lid) are obtained from the above 
formula ( 8 ) . 



25 



. (11a) 
.(lib) 
.(11c) 



-(lid) 



When solving the above formulas (11a) to {116.), the 
above formula (2) corresponding to the above formula (9) 
5 is derived and the basic spectrum X24o.o of the second and 
the fourth quadrants is obtained. 

The inverse Fourier transform circuit 54 performs 
inverse Fourier transform on the image signal S53 input 
from the basic spectrum calculation circuit 53 to obtain 
10 a band broadened digital signal S8 . 

The camera signal processor 9, taking account of the 
correlation of the colors, generates an interpolated RGB 
signal or a luminance signal and a color difference 
signal from R, G, B data included in the image signal S8 
15 from the signal processor 8- 

Below, the operation of an electronic still camera 
of the present embodiment shown in Fig. 2 will be 
explained. 

The CCD image sensor 4 moves in a predetermined 
2 0 direction by an exactly predetermined distance by being 
driven by the CCD drive 6. As shown in Fig. 3, it 
therefore captures four images having mutually different 
sampling phases and outputs received light signals S4o to 
S43 based on the captured four images to the AD converter 
25 5. 
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Next, the AD converter 5 converts the received light 
signals S4o to S43 to digital image signals SSq to 863 and 
outputs them to the signal processor 8. The image signals 
S5o to SSg are stored in the memory 7 , then read and 
5 processed in the signal processor 8 . 

In the signal processor 8, first, the interpolation 
circuit 50 interpolates the input image signals SSq to 
SSg and up- samples them two-fold to generate the image 
signals S50o to S5O3. 

10 Then, the Fourier transform circuit 51 performs two- 

dimensional Fourier transform on the image signals S500 
to S503 to generate the image signals S510 to S513 of the 
frequency domain expression. 

Then, the spatial shift circuit 52 multiples the R 

15 data and B data of the image signals S510 to S513 with 
exp(-j{co^ a^o + ayo)/(2n)) to exp(-j(co^ + cOy 

cXy3)/{2n)) to generate the image signals Yo to Y3. 

The basic spectrum calculation circuit 53 
uses the input image signals Yp to Y3 and the complex 

20 numbers stored in the memory 7 shown in Fig. 2 to 

separately generate an image signal S53 composed of a 
two-dimensional image signal Xq o of the R data and B data 
and outputs the image signal S53 comprised of the 
generated image signal Xq.o of the R data and B data and 

2 5 the G data of the pixels at positions shown in Fig. 4A to 
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the Inverse Fourier transform circuit 54. 

The Inverse Fourier transform circuit 54 performs an 
inverse Fourier transform on the input image signal S53 
to broaden the band and generate a digital signal S8b. 
5 The camera signal processor 9 , taking account of the 

correlation of colors, processes the R, G, B data 
Included in the image signal S8b to generate an 
interpolated RGB signal or a luminance signal and color 
difference signal. 
10 Then, it outputs an image based on the generated RGB 

signal or the luminance signal and color difference 
signal. 

As explained above, according to the electronic 
still camera of the present embodiment, it is possible to 

15 obtain a high definition image signal corresponding to 

double the numbers of pixels of the numbers of pixels of 
the CCD image sensor 4 . 

Also, according to the electronic still camera of 
the present embodiment, although the processing of the 

2 0 signal processor 8 becomes more complicated compared with 
that of the related art, as explained above, since it is 
not necessary to process the G data In the signal 
processor 8, the amount of data processing in the signal 
processor 8 can be reduced compared with a case where all 

25 the R, G and B data has to be processed. 
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Second Embodiment 

Figure 7 Is a view of a part of the configuration of 
an electronic still camera of the present embodiment . 

As shown In Fig. 7, the electronic still camera of 
5 the present embodiment is the same as that of the above 

explained first embodiment except for the processing in a 
signal processor 68. 

Figure 8 is a view of the configuration of the 
signal processor 68 shown in Fig. 7. 
10 As shown in Fig. 8, the signal processor 68 

comprises an interpolation circuit 70, a spatial shift 
circuit 71, and a broad band signal generation circuit 
72. 

The interpolation circuit 70 Interpolates the image 
15 signals SBq to SSg of the four captured images having 

different sampling phases shown in Fig. 3 input from the 
AD converter 5 shown in Fig. 7 to generate the image 
signals S70o to S7O3. 

The spatial shift circuit 71 shifts the image 
20 signals S70o to S7O3 input from the interpolation circuit 
70 in the x-dlreotlon and y-directlon by exactly the 
amount of the phase difference to generate image signals 
yo to Y3 in order to express the fact that the Input 
image signals S70o to S7O3 were sampled at the positions 
2 5 in Figs. 4B and 4C. 
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Specifically, the spatial shift circuit 71 removes 
some samples from the top of the image signals SVOg to 
S7O3 increased in the number of samples or adds some 
samples of appropriate values at the top to generate the 
5 signals Yq to y^. 

The spatial shift circuit 71 outputs the signals Yq 
to Ya to the broad band signal generation circuit 72. 

The broad band signal generation circuit 72 divides 
the image signals Yo to Ys to components wherein both x 
10 and Y have positive frequencies and components wherein 

one of X and y has a negative frequency and the other has 
the opposite positive frequency. Namely, it applies two- 
dimensional filtering for taking out frequencies wherein 
both X and y are positive and a two-dimensional filter 
15 for taking out frequencies wherein one of x and y has a 
negative frequency and the other has the opposite 
positive frequency. The signals obtained in this way are 
Yl3i, and Y24:,. 

The broad band signal generation circuit 72 applies 
20 two-dimensional Hilbert transform on the image signals 

ylSi, and y24i.- Namely, it shifts the phase by n/2(rad) by 
a not shown phase shifting means. The thus obtained 
signals are respectively called ylSt' and y24L'. 

Then, the broad band signal generation circuit 72 
2 5 uses real numbers indicating the real part and imaginary 



part of the complex number read from the memory 7 for 
yl3L and processes [Re(wL) yl3i.+Im(Wi.) yl3i,' ] - In the same 
way, it uses the real numbers indicating the real part 
and imaginary part of the complex number \t^' read from 
5 the memory 7 for y24i, and processes [Re (Wj, ' )y24i,+Im(Wi, ' ) y 
24^' ] . 

Here, Re(x) indicates the real part and Im(x) 
indicates the imaginary part. 

The broad band signal generation circuit 72 adds the 
10 results of [Re(Wi.)yl3i,+Im(Wi.)yl3i.' ] and [ReCwi.' ) y24L+Im(wL 
')y24L'] to cancel (remove) the aliasing to broaden the 
band and obtain the image signal S68. 

Note that in the pre:^ent embodiment , the real 
numbers indicating the real part and imaginary part of 
15 the complex numbers wL and wL ' are stored in the memory 7 
in advance . 

The same effects can be obtained by the electronic 
still camera of the present embodiment as by the above 
explained electronic still camera of the first 
20 embodiment. 

Third Embodiment 

An electronic still camera of the present embodiment 
will be explained with reference to the case of using a 
CCD image sensor having a color compensating filter 
25 instead of a Bayer primary color filter array. 
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Figure 9 is a view of a part of the configuration of 
an electronic still camera of the present embodiment - 

As shown in Fig. 9, the electronic still camera 1 
comprises a lens 2, an optical LPF 3, a CCD image sensor 
5 84, an AD converter 5, CCD drive 86, a memory 87, a 
signal processor 88, and a camera signal processor 9. 

[CCD image sensor 84] 

The CCD image sensor 84 comprises a plurality of 
photodiodes arranged in a matrix and a color compensation 

10 filter. The color compensation filter comprises filters 
transmitting Cy (cyan) color, M (magenta) color, Y 
(yellow), color and W (white) color arranged in the 
pattern shown in Fig. lOA. A photodiode receives light 
passing through the color compensation filter, converts 

15 the result from light to an electrical signal to generate 
received light signals S840 to S843 based on the amount 
of light received, and outputs the received light signals 
S84 to the AD converter 5. 

The CCD image sensor 84, as will be explained later 

20 on, is driven to move by the CCD drive 86, captures four 
images having mutually different sampling phases, and 
generates received light signals S84o to S843 in order to 
obtain a high resolution image shown in Fig. lOB. At this 
time, the sampling phases of the four images may be any 

25 phases. In the present embodiment, for exeimple as shown 
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±n Fig. IOC, they are (0, 0), (n/2, 0), (0, n/2), and 
(n/2, n/2). 

[AD converter 85] 

The AD converter 85 converts the received light 
5 signals S84o to S843 to digital image signals S85o to SSSg 
and outputs the image signals S85o to S853 to the signal 
processor 88. 

[CCD drive 86] 

The CCD drive 86 physically moves the CCD image 
10 sensor 84 in a predetermined direction by an exactly 
predetermined distance based on a control signal S88a 
from the signal processor 88 so as to give a sampling 
interval of the CCD image sensor 84 of 2n and so that the 
CCD image sensor 84 captures a total of four images 
15 having the four different sampling phases shown in Fig. 
IOC. 

[Signal processor 88] 

The signal processor 88 uses the image signals S850 
to S853 of the four captured images having the mutually 
20 different sampling phases shown in Fig. IOC input from 
the AD converter 85 to generate an image signal S88b of 
an image wherein the numbers of pixels in the vertical 
and horizontal directions are doubled. 

Also, the signal processor 88 generates a control 
2 5 signal S88a based on information regarding a direction of 
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movement and distance of the CCD image sensor 84 stored 
in the memory 87 in advance so that the four captured 
images having mutually different sampling phases shown in 
Fig. IOC can be obtained and outputs the control signal 
5 S88a to the CCD drive 86. 

The signal processor 88 writes the image signals 
S850 to S853 of the four captured images of the pixel 
array pattern shown in Fig. lOA input from the AD 
converter 85 to the memory 87, then reads them and 
10 performs the later explained band broadening to generate 
an image signal S88b of a pixel array pattern shown in 
Fig. lOB. 

Below, the processings for generating Cy data. Ye 
data, W data, and G data of the image signal S88b in the 
15 signal processor 88 will be explained. 

Note that the processings for generating the Cy 
data. Ye data, W data, and G data are the same except for 
the values of the sampling phases . 

Here, the sampling phase, as shown in Fig. IOC, has 
2 0 as its origin the position of a pixel outputting a color 
in accordance with the Cy data 

Specifically, for the Cy data, the sampling phases 
of the four captured images, as shown in Fig. IOC, 
respectively become Ca^o' oCy„) = {0, 0), (a^i' ayi) = {n/2, 0), 
25 (a^2' ay2) = {n/2, n/2) , and (of^g- O(y3) = (0, n/2). 
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For the Ye data, the sampling phases of the four 
captured images respectively become (a^o' ayo) = (n/2, 0), 
(Wxi' o<yi) = (n, 0), (a^2' cXy2) = {n/2, 0), and {a^^- o(y3) = (n, 
n/2) . 

5 For the W data, the sampling phases of the four 

captured images respectively become (a^o' «yo) = (0' n/2), 

ayi) = (n/2, n/2), (a^^- ay2) = (0, n/2), and (a^j- 
cXy3) = (n/2, n). 

For the G data, the sampling phases of the four 
10 captured images respectively become (a^o' «yo) = {n/2, n/2), 
(a^i, o(yi) = (n, n/2), (a^^' ofy2) = (n/2, n), and {a^^- ofy3) = (n, 
n). 

The values of these sampling phases are used as 
(a^o' «Yo) to (c<x3' «y3) time of the later explained 

15 spatial shifting and are stored in the memory 87 shown in 
Fig. 9 in advance. 

Figure 11 is a view of the configuration of the 
signal processor 88. 

As shown in Fig. 11, the signal processor 88 
20 comprises an interpolation circuit 90, a Fourier 

transform circuit 91, a spatial shift circuit 92, a basic 
spectrum calculation circuit 93 and an inverse Fourier 
transform circuit 94. 

The interpolation circuit 90 receives as inputs 
25 image signals S85o to SSSg, interpolates and up- samples 
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these two-fold to generate image signals S90o to S9O3, 
and outputs the Image signals S90o to S9O3 to the Fourier 
transform circuit 91. 

The Fourier transform circuit 91 performs a two- 
5 dimensional Fourier transform on the image signals S90o 
to S9O3 input from the Interpolation circuit 90 to 
generate image signals S91o to S9I3 of the frequency 
domain expression and outputs the image signals S91o to 
S9I3 to the spatial shift circuit 92. 

10 The spatial shift circuit 92 performs two- 

dimensional spatial shifting of the Cy data. Ye data, W 
data, and G data included in the image signals S9lo to 
S9I3 obtained by the imaging to generate the image 
signals Yo to Y3 in order to express that sampling was 

15 performed at the above explained positions (a^o' oiyo) to 

Namely, the spatial shift circuit 92 multiplies the 
Cy data. Ye data, W data, and G data of the image signals 
S91o to S9I3 with exp(-j(co^ oc^o+^y O£yo)/(2n)) to exp(-j(co^ 
20 a^j+coy ay3)/(2n)), respectively, to generate the image 
signals Yo to Y3. 

Note, in this case, the portion higher than the 
Nyquist frequency indicates negative frequencies both in 
the x-direction and y-direction. or cOy in that portion 
25 have to be made ((Oj^-cOg^) or (cOy-cOgy) . Here, co^^ and co^y 
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respectively are sampling frequencies In the x- direction 
and. y- direction before up -sampling. 

The basic spectrum calculation circuit 93 uses the 
image signals Yo to Y3 input from the spatial shift 
5 circuit 92 and complex numbers stored in the memory 87 
shown in Fig. 9 to separately generate two-dimensional 
image signals of the Cy data. Ye data, W data, and G 

data and outputs to the Inverse Fourier transform circuit 
94 an image signal S93 comprised by the generated image 

10 signals Xoo of the Cy data. Ye data, W data, and G data. 

Note that the memory 87 stores, for the Cy data. Ye 
data, W data, and G data generated as will be explained 
later on, complex numbers Wo to W3 to be multiplied with 
the image signals Yo to Y3 for obtaining the basic 

15 spectrum components XlSo.o in the first and third 

quadrants and complex numbers Wq' to W3' to be multiplied 
with the image signals Yp to Y3 for obtaining the basic 
spectrum components X24o „ of the second and fourth 
quadrant s . 

20 The basic spectrum calculation circuit 93 obtains 

basic spectrum components X13o,o based on the formula (13) 
below obtained from the formulas {12a to 12d) below and 
obtains basic spectrum components X24o „ based on the 
formula (15) below obtained from the formulas (14a to 

25 14d) below. 
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Yo=Xo.o+Xi,o+Xo.i+Xi,i . . . ( 12a) 

Yi=Xo.o-3Xi,o+Xo,i- jXi i . . . {12b) 

Y2=Xo.o+Xi.o- jXo,i+ jXi i . . . { 12o) 

Y3=Xo,o- jXi,o- jXo,i.-Xi,i . . .{126.) 

5 

X13o.o={- jYo+Yi+Y2+3Y3}/2 ... (13) 

Yo=Xo.o+Xi.o+Xo.i+Xi,i . . . ( 14a) 

Yi=Xo,o+JXi,o+Xo,i.+3Xi 1 (14b) 

10 Y2=Xo „+X,.o-jXo.,-DX,,i ...(14o) 

Y3=Xo.o+dXi,o- jXo,i+Xi,i (14d) 

X24o.o=(Yo- jY^+jY^+jYgl/a ... (15) 



15 Then, as shown In the formula (16) below, the basic 

spectrum components XIS,, o 3.n€L X24o,o are added to generate 
the basic spectrum components Xoo- 



Xo,o=X13o,o+X24o,o ...(16) 

20 

Also, the basic spectrum calculation circuit 93, in 
the same way as for the Cy data, obtains basic spectrum 
components XIS^ o and basic spectrum components X24o „ for 
the Ye data, W data, and G data, and adds them to 
25 generate basic spectrum components Xg q. 
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The Inverse Fourier transform circuit 94 performs an 
inverse Fourier transform on the image signal S93 input 
from the basic spectrum calculation circuit 93 to broaden 
the band and obtain the digital signal S88b. 
5 The camera signal processor 89 processes the Cy 

data. Ye data, W data, and G data included in the image 
signal S88b from the signal processor 88 to generate an 
interpolated RGB signal included or generate a luminance 
signal and color difference signal taking into account 
10 the correlation of the colors. 

Next, the operation of the electronic still camera 
of the present embodiment shown in Fig. 9 will be 
explained . 

The CCD image sensor 84 is driven by the CCD drive 
15 86 to move in a predetermined direction by exactly a 
predetermined distance, captures four images having 
mutually different sampling phase as shown in Fig. IOC, 
and outputs the received light signals S84(, to S843 based 
on the four captured images to the AD converter 85. 
20 Next, the AD converter 85 converts the received 

light signals S84o to S843 to digital image signals S85o 
to S853 and output them to the signal processor 88. The 
image signals S85o to 8863 are stored in the memory 87, 
then read and processed in the signal processor 88. 
2 5 In the signal processor 88, first the interpolation 
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circuit 90 interpolates and up- samples two- fold the input 
image signals S85o to SSSj to generate the image signals 
S90o to S9O3. 

Then, the Fourier transform circuit 91 applies a 
5 two-dimensional Fourier transform to the image signals 

S90o to S9O3 to generate image signals S91o to S9I3 of the 
frequency domain expression. 

Then, the spatial shift circuit 92 multiplies the Cy 
data. Ye data, W data, and G data of the image signals 
10 S91o to S9I3 with exp(-j(co^ a^o + «y cXyo)/(2n)) to expC-j(co^ 
0(^3 + «y C(y3)/(2n)) to generate the image signals Yq to Y3 . 

The basic spectrum calculation circuit 93 uses 
the input image signals Y^ to Yg and the complex numbers 
stored in the memory 87 shown in Fig. 9 to separate 
15 generate two-dimensional image signals Xq.o of the Cy 

data. Ye data, W data, and G data and outputs an image 
signal S93 comprised of the generated image signals Xqo 
of the Cy data. Ye data, W data, and G data to the 
inversed Fourier transform circuit 94. 
20 The inverse Fourier transform circuit 94 applies an 

inverse Fourier transform to the input image signal S93 
to broaden the band and generate the digital signal S88b. 

The camera signal processor 89 processes the Cy 
data. Ye data, W data, and G data included in the image 
2 5 signal S88b to generate an interpolated RGB signal or 



generate a luminance signal and color difference signal 
taking into account the correlation of the colors . 

Then, it outputs an image in accordance with the 
generated RGB signal or the luminance signal and color 
5 difference signal. 

As explained above, according to the electronic 
still camera of the present embodiment, even when using a 
CCD image sensor 84 having the above color compensation 
filter, a high definition image signal corresponding to 
10 double the number of pixels of the CCD image sensor 84 
both in the vertical and horizontal directions can be 
obtained without complicated processing in the camera 
processor 89. 

The present invention is not limited to the above 
15 embodiments. 

For example, the orders of processing in the signal 
processor 8 shown in Fig. 2, the signal processor 68 
shown in Fig. 7 and the signal processor 88 shown in Fig. 
9 are not specifically limited to the above. 
20 Also, the hardware configuration is not limited to 

that in Figs. 2, 7, and 9. 

Also, in the present invention, processing in a 
spatial domain and processing in a frequency domain can 
be mixed together. 
2 5 Furthermore, in the above embodiments, the CCD image 



sensors 4 and 84 were physically moved to obtain four 
captured images having different sampling phase, however, 
a plurality of captured images having different sampling 
phases may be obtained by for example optically or 
5 electrically changing a light path by using a 

birefringent plate without physically moving the CCD 
image sensor . 

Also, the values of the sampling phases are not 
limited to those described in the embodiments . 

10 Also, a CCD image sensor was shown as an example of 

an imaging means in the above embodiments, however, a 
CMOS sensor etc. may also be used. 

Also, in the above embodiments, a case where four 
captured images are used for obtaining a high resolution 

15 image (broadened band) doubled in the vertical and 

horizontal directions was shown as an example, but the 
increase in resolution is not particularly limited to 
two. Further, the increase in resolution may be different 
between the vertical direction and in the horizontal 

20 direction. Also, the band broadening may be not two 
dimensional, but also one dimensional. 

Also, in the above embodiments, four captured images 
were used in order to obtain a high resolution image, 
however, the number of captured images used is not 

25 limited to four since it depends on the multiple in each 
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directions . 

Also, the arrays of the colors of the color filters 
shown in Fig. lA and Fig. lOA can be any arrays. 

The phase shift means used in the spatial domain is 
5 not limited to the Hilbert transform and the amount of 
phase shift is not limited to n/2. 

Summarizing the effect of the invention, as 
explained above, according to the signal processing 
apparatus and the method of the present invention, a 
10 signal can be broadened in band by simple processing. 

Also, according to the image processing apparatus 
and the method of the present invention, a high 
resolution image can be obtained from a low resolution 
image by simple processing. 
15 While the invention has been described with 

reference to specific embodiment chosen for purpose of 
illustration, it should be apparent that numerous 
modifications could be made thereto by those skilled in 
the art without departing from the basic concept and 
20 scope of the invention. 
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What Is claimed Is : 

1. A signal processing apparatus, comprising: 
a transforming means for transforming a 

plurality of first digital signals having mutually 
5 different sampling phases to a plurality of second 
digital signals in a frequency domain; 

a memory means for storing a plurality of 
complex numbers corresponding to said sampling phases; 
and 

10 a processing means for multiplying said 

plurality of second digital signals and said plurality of 
complex numbers corresponding to the plurality of second 
digital signals and adding the multiplied results to 
generate a third digital signal free from aliasing 

15 components. 

2. A signal processing apparatus as set forth in 
claim 1, further comprising: 

a phase shift means for shifting the phase of 
said first digital signals or said second digital signals 
20 by exactly a predetermined phase. 

3. A signal processing apparatus, comprising: 

a phase shift means for shifting the phase of a 
plurality of first digital signals having mutually 
different sampling phases to generate second digital 
25 signals; 
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a memory means for storing a plurality of real 
numbers indicating real parts and Imaginary parts of a 
plurality of predetermined complex numbers corresponding 
to said sampling phases; 
5 a processing means for multiplying said first 

digital signals with real numbers indicating said real 
parts corresponding to the first digital signals to 
obtain first multiplied results, multiplying said second 
digital signals corresponding to said first digital 

10 signals with real numbers indicating said imaginary parts 
corresponding to the second digital signals to obtain 
second multiplied results , and adding said first 
multiplied results and said second multiplied results to 
generate third digital signals free of aliasing 

15 components. 

4. An image processing apparatus, comprising: 

an image input means for generating a plurality 
of first image signals having mutually different sampling 
phases in accordance with imaging results; 

20 a transforming means for transforming the first 

image signals to a plurality of second image signals in a 
frequency domain; 

a memory means for storing a plurality of 
complex numbers corresponding to said sampling phases; 

2 5 a processing means for multiplying said 
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plurality of second Image signals with said plurality of 
complex numbers corresponding to the plurality of second 
image signals and adding the multiplied results to 
generate third image signals free from aliasing 
5 components . 

5 . An image processing apparatus as set forth in 
claim 4, wherein: 

said image input means forms an image of a 
plurality of color lights passed through a single-plate 

10 type color filter on corresponding pixels cunong a 
plurality of pixels arranged in a matrix two- 
dimensionally to generate said first image signals 
comprised by color data of said plurality of colors; and 
said processing means performs said 

15 multiplication and said addition for every color data of 
said plurality of color data to generate a plurality of 
fourth image signals corresponding to said plurality of 
colors and generates said third image signals by using 
the plurality of fourth image signals. 

20 6. An image processing apparatus as set forth in 

claim 5 , wherein : 

said sampling phase is determined for a 
predetermined one color among said plurality of colors in 
order that a sampling pattern of color data of the color 

25 included in said plurality of first image signals and a 
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sampling pattern of color data included in said fourth 
image signals of the color become similar. 

7 . An image processing apparatus as set forth in 
claim 4, wherein said processing means comprises: 

a spatial shift means for spatially shifting 
said second image signals in accordance with said 
sampling phases; 

a basic spectrum calculation means for 
multiplying said spatially shifted plurality of second 
image signals with said plurality of complex numbers 
corresponding to said plurality of second image signals 
and adding the multiplied results to calculate a basic 
spectrum free from aliasing components; and 

an inverse transforming means for transforming 
said basic spectrum from a frequency domain to a time 
domain to generate said third image signals. 

8 . An image processing apparatus as set forth in 
claim 4, further comprising: 

a drive means for moving said image input means 
physically, optically, or electrically so that said 
imaging means can generate a plurality of image signals 
having mutually different sampling phases in accordance 
with the imaging results. 

9 . An image processing apparatus as set forth in 
claim 5, wherein; 
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said linage Input means Is a single- element CCD 
Image sensor and 

said color filter Is a primary color filter or 
a color compensation filter. 
5 10. An image processing apparatus, comprising: 

an image input means for receiving as input a 
plurality of first image signals having mutually 
different seunpling phases in accordance with imaging 
results ; 

10 a phase shift means for shifting the phase of 

said plurality of first digital signals to generate 
second digital signals; 

a memory means for storing a plurality of real 
numbers respectively indicating real parts and Imaginary 

15 parts of a plurality of predetermined complex numbers 
corresponding to said sampling phases; 

a processing means for multiplying said first 
digital signals with real numbers indicating said real 
parts corresponding to the first digital signals to 

20 obtain first multiplied results, multiplying said second 
digital signals corresponding to said first digital 
signals with real numbers indicating said imaginary parts 
corresponding to the second digital signals to obtain 
second multiplied results, and adding said first 

2 5 multiplied results and said second multiplied results to 
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generate third digital signals free from aliasing 
components . 

11. A signal processing method comprising the steps 

of: 

5 transforming a plurality of first digital 

signals having mutually different sampling phases to 
domain a plurality of second digital signals in a 
frequency domain; 

multiplying said plurality of second digital 
10 signals with a plurality of complex numbers corresponding 
to the plurality of second digital signals; and 

adding the multiplied results to generate third 
digital signals free from aliasing components - 

12. A signal processing method using a plurality of 
15 real numbers indicating real parts and imaginary parts of 

a predetermined plurality of complex numbers 
corresponding to seunpling phases, comprising the steps 
of: 

shifting a plurality of first digital signals 
20 having mutually different sampling phases by 

predetermined phases to generate second digital signals; 

multiplying said first digital signals with 
real numbers indicating said real parts corresponding to 
the first digital signals to generate first multiplied 
25 results; 
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multiplying said second digital signals 
corresponding to said first digital signals with real 
numbers Indicating said Imaginary parts corresponding to 
the second digital signal to generate second multiplied 
results; and 

adding said first multiplied results and said 
second multiplied results to generate third digital 
signals free of aliasing components. 

13. An Image processing method comprising the steps 

of: 

generating a plurality of first Image signals 
having mutually different Scunpling phases in accordance 
with imaging results; 

transforming the first image signals to a 
plurality of second image signals in a frequency domain; 
and 

multiplying said plurality of second image 
signals with a plurality of complex numbers corresponding 
to the plurality of second Image signals and adding the 
multiplied results to generate third image signals free 
from aliasing components . 

14 . An image processing method as set forth in 
claim 13 , further comprising the steps of : 

forming an image of a plurality of color lights 
passing through a single -plate type color filter on 
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corresponding pixels among a plurality of pixels arranged 
in a matrix two-dimensionally to generate said first 
image signals comprised by color data of said plurality 
of colors and 

5 performing said multiplication and said 

addition for every color data of said plurality of colors 
to generate a plurality of fourth image signals 
corresponding to said plurality of colors and generate 
said third image signals by using the plurality of fourth 
10 image signals. 

15 . An image processing method as set forth in 
claim 14, further comprising the steps of determining 
said sampling phase for a predetermined one color among 
said plurality of colors so that a samipling pattern of 

15 color data of the color included in said plurality of 

first image signals and a sampling pattern of color data 
included in said fourth image signals of the color become 
similar . 

16. An image processing method using a plurality of 
20 real numbers prepared in advance indicating real parts 

and imaginary parts of a predetermined plurality of 
complex numbers corresponding to sampling phases, 
comprising the steps of : 

generating a plurality of first image signals 
2 5 having mutually different sampling phases in accordance 



with. Imaging results; 

shifting by a predetermined phase said 
plurality of first digital signals to generate second 
digital signals; 
5 multiplying said first digital signals with 

real numbers indicating said real parts corresponding to 
the first digital signals to generate first multiplied 
results ; 

multiplying said second digital signals 
10 corresponding to said first digital signals with real 

numbers indicating said imaginary parts corresponding to 
the second digital signals to generate second multiplied 
results ; and 

adding said first multiplied results and said 
15 second multiplied results to generate third digital 
signals free from aliasing components . 
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SIGNAL PROCESSING APPARATUS. METHOD OF THE SAME, 
AN IMAGE PROCESSING APPARATUS AND METHOD OF THE SAME 

5 ABSTRACT OF THE DISCLOSURE 

An image processing apparatus capable of obtaining a 
high resolution image from a low resolution image without 
making processing in a camera signal processor 

10 complicated, wherein a CCD image sensor is moved by a CCD 
drive and a plurality of received light signals having 
mutually different sampling phases are generated in 
accordance with imaging results in the CCD image sensor, 
the received light signals are converted to image signals 

15 in an AD converter, the image signals are transformed to 
a frequency domain signal in a signal processor and 
multiplied with a plurality of respectively corresponding 
complex numbers stored in a memory, and the multiplied 
results are added to generate an image signal free from 

20 aliasing components. 
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prior United States or PCT International application in the 
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Title 37, Code of Federal Regulations, Section 1 .56 which 
became available between the filing date of the prior 
application and the national or PCT International filing date of 
application. 
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